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Abstract 

Evolutionary  acquisition  holds  the  potential  to  improve  both  the  cost  of  defense 
acquisition  and  the  performance  of  acquired  systems.  Traditional  acquisition  programs 
tend  to  employ  promising,  yet  immature,  technologies  and  develop  them  within  the 
program.  Because  immature  technologies  are  inherently  risky,  unforeseen  obstacles  to 
development  can  lead  to  substantial  cost  overruns  and  schedule  delays.  This  results  in 
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infrequent,  but  large,  increments  of  deployed  capability.  In  contrast,  evolutionary 
acquisition  employs  more  mature,  less-risky  technologies.  This  results  in  more  frequent, 
smaller  increments  of  deployed  capability.  In  theory,  evolutionary  acquisition  could  be 
more  cost  effective  than  traditional  acquisition  approaches  because  it  avoids  most  of  the 
risk  inherent  to  technology  development.  However,  there  is  a  latent  issue  regarding 
evolutionary  acquisition.  If  technology  is  not  matured  within  a  program,  it  must  be 
matured  somewhere  else.  For  critical,  DoD-specific  technologies,  this  cost  must 
logically  fall  on  the  DoD  itself.  The  question,  then,  is  whether  it  is  more  cost  effective  to 
mature  technologies  within  the  R&D  system  or  within  an  acquisition  program?  A 
simulation  of  the  defense  acquisition  system  is  developed  to  address  this  question. 

1.  Introduction 

Over  the  past  several  years,  the  United  States  Department  of  Defense  (DoD)  has 
been  attempting  to  transform  itself  from  an  organization  designed  to  meet  the  Cold  War 
threat  of  the  Soviet  Union  to  a  more  flexible,  adaptable  organization  that  is  ready  to  meet 
the  regional  and  asymmetric  threats  the  US  expects  to  face  in  the  coming  years.  To 
facilitate  this  transformation,  several  modifications  have  been  made  to  the  defense 
acquisition  system — the  most  important  being  the  shift  to  evolutionary  acquisition. 

Evolutionary  acquisition  is  an  attempt  to  address  one  of  the  most  common 
criticisms  of  the  defense  acquisition  system.  Traditional  acquisition  programs  attempt 
large,  revolutionary  leaps  in  system  capability  through  the  use  of  immature  and  risky 
technology.  Not  only  does  immature  technology  often  require  more  time  and  money  to 
develop,  but  it  also  introduces  uncertainty  that  may  lead  to  significant  delays  and  cost 
overruns.  Consequently,  warfighters  must  often  make  due  with  increasingly  obsolete 
equipment  during  the  long  intervals  between  new  system  deployments,  and  there  is  little 
flexibility  to  adapt  to  emerging  threats  and  exploit  technology  opportunities. 

Evolutionary  acquisition,  on  the  other  hand,  attempts  to  set  more  modest 
capability  goals  for  each  acquisition.  The  idea  is  to  use  more  mature,  and  hence,  less- 
risky  technology,  in  order  to  shorten  acquisition  cycle-times.  Thus,  each  acquisition 
cycle  under  evolutionary  acquisition  should  be  shorter  and  cost  less  than  more 
traditional  programs.  As  a  result,  warfighters  should  receive  more  frequent  upgrades  to 
their  equipment  and,  thus,  should  be  at  less  risk  of  going  to  war  with  obsolete  hardware. 

Despite  the  apparent  motivation  to  implement  evolutionary  acquisition  and 
committing  the  approach  to  policy,  it  would  seem  that  the  DoD  has  had  limited  success 
in  doing  so  (Lorell,  Lowell  &  Younossi,  2006).  In  fact,  the  US  Government  Accountability 
Office  (GAO)  has  suggested  that  DoD  reforms  have  not  gone  far  enough  (GAO  2003; 
2006,  September;  2006,  April;  2007a).  They  advocate  adapting  commercial  best 
practices  regarding  technology  and  product  development  to  the  defense  acquisition 
system.  Among  these  are  a  centralized  portfolio  approach  to  managing  new  systems 
and  technologies,  a  staged  knowledge-based  approach  to  both  acquisition  and 
technology  development,  strict  enforcement  of  technology  maturity  requirements,  and  a 
more  evolutionary  approach  to  new  system  development. 
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Since  these  reforms  are  derived  from  the  commercial  world,  the  obvious  question 
is  whether  they  will  translate  well  to  a  government  context.  The  defense  acquisition 
system  differs  from  a  commercial  product  development  process  in  several  respects.  In 
particular,  the  government  essentially  serves  as  a  technology  developer,  system 
developer,  customer,  and  user.  Furthermore,  the  DoD  and  a  few  allies  are  really  the 
only  customers  for  the  systems  and  technologies  developed  within  the  defense 
acquisition  system.  Thus,  there  is  a  more  limited  capacity  to  purchase  multiple 
evolutionary  iterations  of  a  system  than  there  would  be  with  a  consumer  product. 
Consequently,  the  pertinent  question  is,  if  evolutionary  acquisition  were  fully 
implemented,  would  there  be  any  tangible  benefit  for  the  Department  of  Defense?  Since 
evolutionary  acquisition  is  inseparable  from  technology  policy,  the  objective  of  this  paper 
is  to  consider  the  implications  of  an  evolutionary  technology  policy  on  the  cost  and 
performance  of  the  defense  acquisition  system. 

As  a  first  step  to  better  understanding  the  system  level  trade-offs  of  technology 
policy  on  acquisition,  the  work  presented  in  this  paper  attempts  to  model  the  basic 
“physics”  of  the  acquisition  system,  in  particular,  the  relationship  between  the  R&D 
process  and  the  acquisition  lifecycle.  The  purpose  is  to  gain  insight  into  the  most 
fundamental  system-level  influences  on  the  efficacy  of  acquisition  policies.  To  that  end, 
an  idealized  view  of  the  acquisition  system  is  adopted  to  which  complicating  factors  may 
be  subsequently  added  to  test  their  effects.  The  acquisition  model  was  implemented  as 
a  discrete  event  simulation  with  the  key  decision  variable  being  the  maturity  level  at 
which  a  technology  moves  from  R&D  to  an  acquisition  effort.  Extensive  sensitivity 
analyses  were  performed  and  several  insights  into  the  impact  of  technology  policy  on 
acquisition  were  generated.  The  most  important  output  of  this  effort,  however,  is  an 
informed  set  of  future  research  directions  that  will  facilitate  more  definitive  answers  to 
major  policy  questions  regarding  evolutionary  acquisition.  What  follows  is  a  summary  of 
key  findings.  For  a  more  detailed  discussion  of  the  analysis  approach  and  results  see 
Pennock  (2008). 

2.  Background 

As  was  mentioned  previously,  evolutionary  acquisition  is  an  attempt  to  reduce 
acquisition  cycle-times  by  setting  capability  goals  that  are  more  modest  than  is  typical  of 
a  traditional  program.  This  allows  programs  to  utilize  more  mature  technology  and, 
hence,  reduce  the  amount  of  technology  risk.  In  theory,  this  should  reduce  cost, 
schedule,  and  performance  uncertainty.  The  hope  is  that  it  will  lead  to  less-expensive 
acquisition  programs  that  proceed  more  quickly.  Consequently,  warfighters  would 
receive  up-to-date  equipment  more  frequently  and  at  lower  cost. 

The  motivating  issue  behind  evolutionary  acquisition  is  cycle-time.  In  theory, 
shorter  cycles  mean  that  each  is  less  expensive  and  new  technologies  can  be  moved 
into  the  field  faster  to  meet  emerging  warfighter  needs.  The  driving  issue,  then,  is  how 
big  of  a  leap  in  capability  should  one  attempt  during  each  acquisition  cycle?  Of  course, 
the  risk  associated  with  the  size  of  the  leap  is  linked  to  the  maturity  of  the  required 
technology.  Thus,  evolutionary  acquisition  is  really  all  about  technology  policy  because 
with  a  large  enough  leap,  evolutionary  becomes  revolutionary. 
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So  where  does  the  DoD's  approach  to  evolutionary  acquisition  come  in?  A  key 
issue  is  that  the  DoD  does  not  manage  technology  or  “product”  portfolios  in  the  same 
manner  as  a  large  commercial  enterprise.  In  part,  this  is  due  the  public  nature  of  the 
defense  enterprise.  Even  so,  the  GAO  asserts  that  the  DoD  should  adopt  additional 
commercial  best  practices  regarding  the  centralized  management  of  its  acquisition  and 
technology  portfolios  and  the  management  of  technology  transitions  from  R&D  to 
acquisition  programs  (GAO,  2006,  September;  2007a).  Under  the  current  system,  there 
is  often  a  funding  gap  in  technology  development.  Early-stage  technologies  are  funded 
through  the  R&D  system  (or  S&T  as  it  is  known  in  the  DoD),  and  late-stage  technologies 
are  often  funded  in  support  of  a  particular  acquisition  effort.  It  is  technologies  in  the 
middle  stages  of  maturation  that  are  often  left  without  obvious  ownership  and  hence 
funding.  Consequently,  if  certain  technologies  are  required  by  an  acquisition  effort,  their 
development  through  the  middle  stages  must  be  funded  in  support  of  the  development  of 
a  particular  system.  This  requires  early  commitment  to  a  technology  when  its  final 
realization  is  still  uncertain.  In  the  past,  this  has  often  led  to  disappointment  as 
technologies  took  longer  to  develop  and  did  not  perform  as  well  as  expected.  In  fact,  a 
recent  National  Research  Council  report  suggests  that  concept  decisions  made  prior  to 
Milestone  A  determine  70-75%  of  lifecycle  costs  (NRC,  2008).  Early  commitment  to 
system  concepts  that  depend  on  immature  technologies  sacrifices  flexibility  and  may 
lead  to  costly  rework. 

Theoretically,  if  the  DoD  adopted  the  commercial  new  product  model  that  the 
GAO  suggests  (GAO,  2006,  September;  2007a),  it  would  allow  the  DoD  greater  flexibility 
in  how  to  select  and  mature  technologies  for  development  in  anticipation  of  future 
acquisition  program  needs.  In  essence,  it  would  lead  to  the  creation  of  additional 
technology  options.  This  would  reduce  the  burden  and  risks  of  technology  development 
on  acquisition  programs  since  they  could  choose  from  a  portfolio  of  mature  technologies. 

So  in  the  end,  the  two  fundamental  questions  of  evolutionary  acquisition  are  how 
mature  should  technologies  be  when  they  are  transitioned  from  R&D  to  acquisition 
efforts,  and  what  is  the  best  approach  to  mature  them?  All  else  being  equal,  this 
essentially  determines  the  acquisition  cycle-time  as  well  as  the  size  of  the  capability 
improvement  for  each  cycle.  Ultimately,  the  answer  will  hinge  on  factors  such  as  the 
cost  of  technology  maturation,  the  rate  of  learning  from  fielded  systems,  and  the 
overhead  cost  associated  with  an  acquisition  cycle. 


3.  Model  Setup 

The  motivation  behind  the  structure  of  the  model  is  to  represent  the  set  of 
commercial  best  practices  recommended  by  the  GAO  for  implementation  in  the  context 
of  the  defense  acquisition  system.  This  includes  both  a  staged,  centrally  managed 
technology  development  process  as  well  a  strictly  enforced  acquisition  program  lifecycle. 
Given  the  staged  nature  of  both  R&D  and  acquisition,  discrete  event  simulation  was  the 
logical  choice  to  capture  the  behavior  of  the  system.  As  was  mentioned  previously,  the 
representation  of  the  defense  acquisition  system  presented  here  is  intentionally  scaled- 
down  and  idealized.  The  benefit  of  an  idealized  model  is  two-fold.  First,  the  scaled-down 
representation  is  more  tractable  and  allows  us  to  attempt  multiple  experimental 
excursions.  Second,  it  allows  us  to  consider  the  structural  impacts  of  technology  policy 
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unobscured  by  the  inconsistent  implementation  that  occurs  in  the  actual  defense 
acquisition  system.  In  particular,  the  modeling  emphasis  was  on  the  linkage  between 
the  movement  of  technologies  through  the  R&D  process  to  the  length  and  cost  of  the 
acquisition  cycles.  In  order  to  represent  the  impact  of  technology  policy  on  defense 
acquisition,  there  were  three  key  features  of  the  system  that  required  consideration:  the 
movement  of  technologies  through  the  R&D  system,  the  movement  of  programs  through 
the  acquisition  process,  and  the  rate  of  technological  progression. 

The  simulation  was  implemented  using  the  Arena  10.0  software  package  and 
consists  of  three  major  components:  the  technology  development  process  model,  the 
system  acquisition  process  model,  and  the  technical  progress  model.  The  technology 
development  process  model  describes  how  technologies  with  potential  defense 
application  are  matured  through  the  defense  R&D  system.  This  process  provides  a 
portfolio  of  technologies  for  use  by  acquisition  programs.  The  system  acquisition 
process  model  describes  the  lifecycle  of  a  defense  acquisition  program  from  concept 
development  to  deployment.  Finally,  the  technical  progress  model  describes  how  the 
capabilities  provided  by  technologies  improve  over  time. 

3.1  Technology  Development  Process  Model 

The  technology  development  process  model  simulates  the  movement  of 
individual  technologies  through  a  maturation  process.  Ideally,  a  technology  development 
process  is  centrally  managed  and  staged.  Technologies  are  selected  for  development 
based  on  their  potential  applicability  to  future  products.  In  the  commercial  world,  product 
and  technology  roadmaps  drive  development.  These  roadmaps,  and  the  organization's 
commitment  to  them,  provide  a  shared  vision  that  the  DoD  often  lacks.  However, 
developing  the  technologies  to  satisfy  the  roadmap  entails  a  certain  amount  of  risk.  In 
order  to  mitigate  that  risk,  each  technology  must  pass  through  a  series  of  stage-gates. 
Each  gate  provides  an  opportunity  to  evaluate  the  status  of  a  technology  and  determine 
whether  it  should  continue  to  receive  funding.  Such  a  system  facilitates  prioritization  of 
technology  projects  as  well  as  risk  mitigation.  It  is  important  to  note  that  the  Department 
of  Defense  has  not  consistently  implemented  such  a  system  (GAO,  2006,  September). 
Instead,  there  are  a  number  of  different  organizations  throughout  the  DoD  that  perform 
or  fund  R&D  work,  each  with  its  own  way  of  managing  technology  projects.  These 
inconsistencies  preclude  the  effective  management  of  technology  development  and 
promote  duplication  and  mismatch  between  the  technology  supplied  by  R&D 
organizations  and  the  technology  demanded  by  acquisition  programs.  Consequently,  for 
this  study,  the  technology  development  process  was  modeled  in  the  spirit  of  the  GAO's 
recommendation  of  a  centrally-managed  and  staged  technology  development  process. 

The  process  starts  when  new  but  immature  technologies  arrive  for  evaluation. 
The  arriving  technologies  are  prioritized  and  then  funded  until  the  budget  is  expended. 
Rejected  technologies  are  considered  for  funding  in  future  rounds,  and  successfully 
matured  technologies  move  on  to  the  next  stage.  The  sequence  repeats  until  each 
technology  is  either  successfully  matured  or  discarded.  The  maturity  of  each  technology 
is  measured  by  the  Technology  Readiness  Level  (TRL)  scale.  The  purpose  of  a 
properly  functioning  technology  development  process  is  to  prioritize  and  fund  these 
technologies  by  potential  cost  and  benefit.  The  process  used  in  this  simulation  is 
represented  in  Figure  1.  For  more  information  see  Pennock  (2008). 
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Figure  1.  The  Technology  Development  Process  Model 

3.2  System  Acquisition  Process  Model 

The  system  acquisition  process  model  describes  the  lifecycle  of  a  defense 
acquisition  program.  It  is  based  on  the  five-stage  Defense  Acquisition  Management 
Framework  (DoD,  2006).  As  with  the  technology  development  process  model  described 
above,  the  acquisition  process  model  in  the  simulation  will  assume  that  acquisition 
programs  follow  the  rules,  and,  consequently,  programs  will  move  through  each  phase  in 
order  with  no  concurrency. 

Within  the  simulation  model,  the  basic  unit  in  the  system  acquisition  process  is  a 
program  to  acquire  a  system.  It  is  assumed  that  the  DoD  has  several  types  of  systems. 
Each  type  is  continuously  cycling  through  the  acquisition  process.  For  example,  if  the 
Air  Force  deploys  a  new  air  superiority  fighter,  it  is  assumed  that  it  will  begin  concept 
development  of  its  replacement  shortly  thereafter.  This  assumption  will  be  relaxed  later. 

Each  type  of  system  is  dependent  upon  several  technologies,  each  from  a 
different  application  area.  For  example,  an  air  superiority  fighter  might  require  a 
propulsion  technology,  a  sensor  technology,  and  an  avionics  technology.  The 
acquisition  process  model  used  in  the  simulation  is  illustrated  in  Figure  2.  For  more 
information  see  Pennock  (2008). 
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Note:  Operations  and  Support  was  considered  outside  the  scope  of  this  simulation. 

Figure  2.  The  System  Acquisition  Process  Model 


Ultimately,  the  purpose  of  acquiring  a  system  is  to  provide  military  capabilities.  It 
is  assumed  that  each  system  deployed  provides  a  capability.  Capability  in  the  model  is 
an  abstract  representation  of  military  utility.  It  is  assumed  that  there  is  a  synergistic 
effect  between  the  technologies  employed  in  the  system:  the  system  being  greater  than 
the  sum  of  its  parts.  Thus,  a  multiplicative  model  is  used  to  represent  capability.  The 
capability  of  a  deployed  system  is  the  product  of  the  performance  levels  for  each  of  its 
required  technologies.  Thus,  an  air  superiority  fighter  without  a  propulsion  system  is 
useless  no  matter  how  capable  its  sensor  is.  This  measure  of  capability  allows  us  to 
determine  the  cost  effectiveness  of  a  particular  technology  policy. 

3.3  Technical  Progress  Model 

The  final  key  feature  of  the  simulation  is  the  model  of  technical  progress.  Where 
do  new,  more  capable  technologies  come  from?  It  is  important  to  note  that  the 
technology  development  model  in  this  simulation  does  not  consider  basic  research.  In 
fact,  TRL  1  signifies  the  transition  of  ideas,  concepts,  and  technologies  from  basic 
research  to  applied  research.  Thus,  we  can  assume  that  there  is  a  certain  amount  of 
research  occurring  exogenous  to  the  simulation.  This  research  may  come  from 
government  or  commercial  sources.  The  key  is  that  there  is  a  constant  inflow  of  new 
technologies  and  that  their  performance  improves  overtime.  The  purpose  of  the 
technology  development  process  is  to  adapt  these  technologies  for  use  in  a  military 
system.  There  is  one  caveat,  however,  and  that  is  that  a  purely  exogenous  technical 
progress  model  neglects  the  learning  that  inevitably  occurs  from  fielding  systems.  For 
example,  valuable  information  gathered  from  field  use  of  a  jet  engine  will  likely  inform  the 
development  of  the  next  generation  jet  engine.  Thus,  there  is  a  learning  effect,  and  the 
more  rapidly  systems  are  fielded,  the  sooner  subsequent  learning  will  be  available  for 
future  technologies.  This  is  especially  true  for  military-specific  technologies  in  which  the 
only  source  of  user  feedback  is  the  military  itself. 

Consequently,  the  technical  progress  model  in  this  simulation  attempts  to  model 
both  these  features.  To  do  so,  a  hybrid  model  was  created.  First,  there  is  a  baseline 
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technology  coefficient  for  each  application  area.  Whenever  a  technology  is  fielded,  the 
coefficient  is  multiplied  by  a  learning  factor  (e.g.,  1.1).  This  captures  the  learning  from 
implementation.  Second,  there  is  an  exponential  growth  model  for  each  application  area 
This  represents  the  learning  from  exogenous  R&D  activities.  The  two  are  multiplied 
together  to  determine  the  current  technology  level  and  are  represented  by  the  equation 


Ces‘ 


where  C  is  the  technology  coefficient  and  g  is  the  exogenous  growth  rate. 
Arriving  technologies  are  assigned  a  performance  as  a  random  variation  on  this  value. 
The  parameters  of  this  model  can  be  adjusted  to  accommodate  the  specific  situation  of 
each  application  area.  For  example,  technologies  that  are  used  commercially  may  have 
a  high  exogenous  growth  rate  and  low  learning  factor  because  their  progress  would 
continue  regardless  of  military  use.  The  reverse  may  be  true  for  military-specific 
technologies  since  there  would  be  little  learning  from  commercial  use. 

4.  Experimental  Design 

4.1  Simulation  Parameters 

As  previously  mentioned,  the  DoD  has  been  relatively  inconsistent  in  its 
implementation  of  its  own  policies,  and  evolutionary  policies — in  particular — are  fairly 
new.  Consequently,  using  historical  data  to  set  simulation  parameters  is  particularly 
problematic.  In  fact,  a  RAND  study  to  assess  cost  growth  in  weapon  system  programs 
found  a  number  of  issues  in  the  available  cost  data  for  defense  acquisition  programs 
(Arena,  Leonard,  Murray  &  Younossi,  2006).  Some  of  these  issues  include  significant 
aggregation  of  data,  baseline  changes,  changes  in  reporting  guidelines,  and  incomplete 
data.  The  situation  is  worse  for  technology  maturation.  As  indicated  by  the  GAO,  the 
DoD  does  not  systematically  track  its  technology  development  efforts  (GAO,  2006, 
September).  Furthermore,  the  introduction  of  TRL  levels  to  the  DoD  is  fairly  recent,  so 
there  is  little  experience  with  their  application  in  a  DoD  context.  Since  NASA  has  been 
using  the  TRL  scale  for  some  time,  it  would  seemingly  be  a  logical  source  of  information 
regarding  the  cost  and  risk  associated  with  maturing  technologies  through  TRL  levels. 
Unfortunately,  a  2005  study  at  NASA  to  determine  the  cost  and  risk  found  that  poor 
record  keeping  resulted  in  insufficient  useful  data  to  achieve  statistically  significant 
results  (Kirn,  2005). 

Fortunately,  the  aim  of  this  study  is  not  to  precisely  recreate  the  defense 
acquisition  system  as  it  is  but  to  identify  policy  directions  to  determine  how  it  should  be. 
This,  in  combination  with  extensive  sensitivity  analysis,  allows  for  a  more  reasonable 
margin  of  error  in  setting  the  simulation  parameters.  Consequently,  the  actual  values 
used  in  the  experiments  are  an  amalgamation  from  several  sources,  including  reports 
and  studies  from  both  government  and  commercial  sources  (Bodner  &  Rouse,  2007; 
DoD,  2007  April;  DoD,  2007  August;  Fox,  1988;  GAO,  2007b;  Kirn,  2005;  Stevens  & 
Burley,  1997). 
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4.2  Basic  Experiment 

In  order  to  answer  the  research  questions  posed  in  this  paper,  three  cases  were 
developed.  The  three  cases  are  variations  on  the  key  experimental  variables,  the  Min 
TRL  and  the  Fallback  TRL.  The  Min  TRL  is  the  minimum  maturity  requirement  for  a 
technology  used  in  an  acquisition  program,  and  the  Fallback  TRL  is  the  minimum 
maturity  selected  when  the  first  choice  technology  fails.  The  cases  are  as  follows: 

•  Base  Case — The  base  case  most  closely  resembles  the  current  modus  operandi  of 
the  defense  acquisition  system.  Technologies  are  selected  at  mid-TRL  levels  and 
final  maturation  occurs  during  the  technology  development  phase  of  an  acquisition 
effort.  High  performing,  but  immature  technologies  are  preferred  over  more  mature, 
proven  technologies.  If  a  technology  fails,  however,  the  program  will  fall  back  to  a 
more  mature  technology. 

o  Min  TRL  =  4 
o  Fallback  TRL  =  7 

•  Evolutionary  Acquisition — In  this  case,  programs  may  only  use  fully  mature 
technology.  Maturation  of  technology  is  funded  in  the  R&D  system,  and  there  is 
effectively  no  technology  development  phase.  (Note  that  TRL  7  was  chosen  here 
because  TRL  levels  8  and  9  are  system  specific.) 

o  Min  TRL  =  7 

•  Revolutionary  Acquisition — Programs  target  maximum  performance  at  all  costs 
and,  thus,  always  choose  the  technologies  with  the  highest  expected  performance. 
When  a  technology  fails,  another  top  performer  is  selected  in  its  place. 

o  Min  TRL  =  4 
o  Fallback  TRL  =  4 


There  are  several  outputs  of  interest.  These  are  the  cost  of  operating  the  entire 
acquisition  system,  the  cost  of  an  individual  program,  the  annual  capability  growth  rate, 
and  the  acquisition  program  length.  Of  course,  we  are  interested  in  the  long-run 
behavior  of  these  outputs.  Consequently,  to  perform  the  experiments,  the  simulation 
was  run  for  a  warm-up  period  in  order  to  fully  populate  the  technology  portfolio,  and  then 
statistics  were  collected  on  the  outputs  of  interest. 

In  particular,  each  simulation  was  run  for  a  warm-up  period  of  50  years  and  then 
statistics  were  collected  for  another  150  years.  There  are  40  replications  for  each 
experimental  case.  As  for  the  acquisition  programs,  there  are  three  system  types  each 
requiring  three  technologies.  Each  of  those  technologies  falls  into  one  of  six  application 
areas.  It  was  assumed  that  the  three  acquisition  programs  are  homogenous  in  terms  of 
cost  and  schedule  risk,  and  it  was  also  assumed  that  the  application  areas  are 
homogeneous  in  terms  of  cost,  schedule,  and  technical  risk.  The  budget  for  the 
technology  development  process  was  set  to  $3  billion,  and  was  allocated  among  the  six 
stages  so  as  to  ensure  a  smooth  flow  of  technologies  through  the  system.  It  was  also 
assumed  that  all  of  the  stages  are  of  equal  length.  This  is  simply  to  focus  on  the 
technical  risk  for  the  basic  experiment.  Finally,  the  technical  progress  model  is  identical 
for  all  six  application  areas  and  features  a  mix  of  exogenous  technical  progression  and 
learning. 
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4.3  Sensitivity  Analysis 

The  simulation  developed  is  quite  flexible  and  many  different  scenarios  can  be 
analyzed.  A  first  order  sensitivity  analysis  was  performed,  and  the  results  are  presented 
in  Pennock  (2008).  It  was  found  that  the  simulation  outputs  were  particularly  sensitive  to 
five  factors:  the  R&D  budget  size,  the  R&D  budget  distribution,  the  rate  of  technical 
learning,  the  technology  development  stage  length,  and  production  costs.  The  impact  of 
the  size  of  the  R&D  budget  was  examined  by  leaving  the  percent  allocated  per  stage  the 
same  but  varying  the  aggregate  amount  over  the  range  of  -50%  to  +50%.  The  budget 
distribution  was  analyzed  by  reducing  the  budget  for  stages  4,  5,  and  6.  This  particular 
scenario  was  designed  to  represent  the  status  quo  of  the  defense  technology 
development  process.  To  understand  the  influence  of  the  rate  of  technical  learning,  the 
learning  factor  from  the  technical  progress  model  was  varied  between  1  (no  learning) 
and  2.  In  the  basic  experiment,  all  technology  development  stages  are  one  year  in 
length.  To  understand  the  impact  of  stage  length,  the  scenarios  were  run  with  stage 
lengths  of  two  years  and  three  years.  Finally,  the  influence  of  production  costs  was 
analyzed  by  varying  the  cost  rate  from  -100%  to  +100%  of  the  baseline  value. 

5.  Results  and  Analysis 

5.1  Results  of  the  Basic  Experiment 

First,  we  will  consider  the  results  of  the  basic  experiment.  The  average  values  of 
each  of  the  output  statistics  are  displayed  in  Table  1 .  Note  that  for  compactness,  system 
specific  outputs  are  only  shown  for  system  1 .  The  results  are  similar  for  the  other  two 
systems.  The  most  obvious  question  is  how  do  these  program  outputs  compare  to  real 
acquisition  programs?  As  far  as  program  duration,  the  distributional  parameters  for 
concept  development,  system  development,  and  production  were  derived  from  Fox 
(1988,  p.  29).  with  an  average  program  duration  of  15  years.  We  see  from  Table  1  that 
the  base  case  has  an  average  duration  of  14  years,  which  is  fairly  close.  As  for  cost, 

Fox  does  not  provide  cost  data,  but  a  recent  GAO  report  provides  the  cost  and  schedule 
performance  of  62  current  weapons  system  programs  (GAO,  2007b).  An  analysis  of 
these  data  reveals  that  the  average  program  cost  is  approximately  $16  billion.  An 
important  caveat  is  that  these  data  cover  a  wide  range  of  programs.  Some  are  small 
upgrade  programs  that  are  short  and  inexpensive  while  others  are  major  system  of 
systems  acquisitions  that  will  take  30  years  and  cost  hundreds  of  billions  of  dollars. 

Even  so,  we  can  see  from  Table  1  that  the  average  program  cost  for  the  base  case  is 
approximately  $16  billion.  Thus,  we  can  say  that  the  simulation  outputs  are  within  the 
right  order  of  magnitude  for  an  “average”  acquisition  program. 
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Table  1 .  The  Average  Output  Values  over  40  Repetitions  for  the  Scenarios 

of  the  Basic  Experiment 


Output 

Base  Case 

Evolutionary 

Revolutionary 

Total  Acquisition  System 
Operating  Cost 

($  million,  annualized) 

5807 

6410 

5169 

Capability  Growth  Rate 

(System  1) 

0.16 

0.179 

0.138 

Program  Duration 

(System  1 ,  years) 

14.3 

11.8 

17.2 

Program  Cost 

(System  1 ,  $  million) 

16091 

14668 

16736 

In  order  to  understand  these  results  fully,  we  will  address  each  of  the  four  outputs 
in  turn.  Figure  3a  depicts  the  95%  confidence  intervals  for  the  average  annual  cost  to 
operate  the  acquisition  system.  Clearly,  evolutionary  acquisition  is  the  most  expensive 
and  revolutionary  acquisition  is  the  least  expensive.  If  the  technology  policy  is  less 
aggressive  with  evolutionary  acquisition,  why  would  it  be  more  expensive?  To  better 
understand  this  outcome,  let  us  consider  the  average  cost  of  the  individual  programs. 

Figure  3b  shows  the  confidence  intervals  for  the  average  program  cost  to  acquire 
a  system  of  type  1 .  Here  we  see  that  the  average  program  cost  is  actually  lower  with 
evolutionary  acquisition  than  revolutionary  acquisition.  So  as  evolutionary  acquisition 
supporters  suggest,  using  mature  technology  must  lower  program  cost.  Then  why  does 
the  acquisition  system  cost  more  to  operate  under  evolutionary  acquisition? 

The  answer  is  revealed  when  we  examine  the  average  program  duration  or 
cycle-time.  In  Figure  3c,  we  see  that  the  program  length  is  much  shorter  with 
evolutionary  acquisition.  With  a  shorter  cycle-time,  acquisitions  happen  more  frequently. 
Each  cycle  imposes  overhead  costs  including  system  development,  production,  and 
deployment  costs.  Since  these  overhead  costs  are  far  greater  than  any  savings  that 
would  result  from  more  efficient  management  of  the  technology  portfolio,  the  overall  cost 
rises. 


But  does  the  additional  cost  of  evolutionary  acquisition  buy  the  DoD  anything? 
Figure  3d  reveals  that  evolutionary  acquisition  results  in  a  superior  annual  capability 
growth  rate.  The  annual  capability  growth  rate  is  the  “average”  annual  rate  of  capability 
improvement.  Much  like  an  interest  rate,  even  small  differences  in  the  rate  can  result  in  a 
huge  difference  in  the  level  of  deployed  capability  over  the  long-run.  Thus,  we  see  that 
there  is  a  cost/performance  trade-off  governed  by  the  technology  maturity  requirement. 
Allowing  less-mature  technology  hurts  system  performance  because  it  takes  longer  to 
move  technologies  into  the  field,  but  since  it  incurs  large  production  costs  less  often,  it  is 
also  less  expensive.  Strictly  enforcing  maturity  requirements,  on  the  other  hand,  means 
shorter,  less-expensive  programs  that  achieve  high  performance  by  moving  technologies 
into  the  field  more  quickly.  Unfortunately,  this  incurs  production  costs  more  frequently 
and  results  in  increased  operating  costs  for  the  acquisition  system  as  a  whole. 
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In  fact,  by  varying  the  technology  policy  we  can  move  along  a  roughly  linear 
frontier  of  cost/performance  combinations.  Figure  4  shows  the  cost  and  performance  for 
all  possible  technology  polices  such  that  1  <  Min  TRL  <  7  and  Fallback  TRL  >  Min  TRL. 
At  first,  this  result  would  seem  to  suggest  that  technology  policy  should  not  be  strictly 
enforced  as  budgetary  restrictions  would  force  changes  in  technology  policy  to  meet  cost 
goals.  Fortunately,  this  is  not  the  case. 

In  order  to  maintain  a  consistent,  evolutionary  technology  policy  but  retain  the 
ability  to  trade  performance  for  cost,  all  that  is  required  is  to  insert  a  delay  between 
acquisition  cycles.  Figure  5  depicts  the  cost/performance  combinations  for  the 
evolutionary  policy  with  inter-cycle  delays  ranging  from  0  to  7  years.  Also  shown  is  the 
linear  trend  line  from  Figure  4.  Clearly,  the  introduction  of  a  delay  allows  the 
evolutionary  policy  to  replicate  the  cost/performance  combinations  achieved  through 
shifts  in  technology  policy.  Thus,  for  any  given  cost  target,  an  efficient  policy  can  be 
found  by  imposing  the  evolutionary  maturity  requirements  in  combination  with  the 
appropriate  inter-acquisition  cycle  delay. 


5.2  Sensitivity  Results 

The  previous  section  presented  the  results  of  the  basic  experiment,  but  there 
remains  a  question  of  robustness.  How  stable  are  results?  Are  there  any  cases  in 
which  the  evolutionary  policy  is  not  the  best  performing?  While  five  scenarios  were 
described  in  Section  4.3  above,  due  to  space  constraints  only  two  will  be  presented 
here.  For  the  remainder  see  Pennock  (2008). 

First,  we  consider  the  distribution  of  the  R&D  budget  among  the  stages.  In 
particular,  this  scenario  is  designed  to  represent  a  situation  that  is  often  referred  to  as 
crossing  the  chasm.  Crossing  the  chasm  describes  the  difficulty  that  technology 
development  efforts  often  encounter  in  moving  through  the  middle  stages  of  technology 
maturation  because  of  a  scarcity  of  funding.  To  simulate  this  scenario,  funding  for 
stages  4,  5,  and  6  was  varied  over  a  range  of  25%  to  100%  of  the  baseline  value. 
Figure  6  reveals  that  the  best  policy  from  a  performance  standpoint  is  quite  sensitive  to 
the  level  of  middle-stage  funding. 
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Figure  3.  95%  Confidence  Intervals  by  Scenario  for  the  Mean  Values  of  Each  of  the  Four  Primary  Outputs 

Note:  The  height  of  the  box  represents  the  width  of  the  confidence  interval,  and  the  vertical  lines  represent  the  range  of  realized  values 


Annual  Operating  Cost  ($  million/year) 


Figure  4.  Cost/Performance  Trade-off  for  All  Possible  Technology  Policies  with  a  Linear 

Trend  Line 
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Figure  5.  Cost/Performance  Trade-off  Replicated  through  the  Evolutionary  Policy  with  an 

Inter-cycle  Delay 
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Figure  6.  Capability  Growth  Rate  when  Middle-stage  R&D  Funding  Is  Cut 


As  we  would  expect,  the  evolutionary  policy  is  the  most  sensitive  since  it  is  dependent 
upon  a  constant  supply  of  mature  technologies.  On  the  opposite  end,  the  revolutionary  policy  is 
the  most  robust  since  it  can  provide  its  own  middle-stage  funding,  and  once  again,  the  base 
case  falls  in  between.  Given  the  varied  rates  of  performance  decay  among  the  three  policies, 
there  are  domains  in  which  each  is  dominant.  When  R&D  is  well  funded,  the  evolutionary  policy 
provides  superior  performance.  As  middle-stage  funding  is  reduced  by  more  than  25%,  the 
performance  of  the  base  case  policy  begins  to  exceed  the  performance  of  the  evolutionary 
policy.  As  funding  declines  further,  the  revolutionary  policy  becomes  the  top  performing  policy. 

Of  all  of  the  scenarios  presented  in  this  paper,  the  crossing  the  chasm  scenario  is 
probably  the  most  similar  to  business  as  usual  at  the  DoD.  Typically,  S&T  funding  covers  early- 
stage  technology  development,  but  once  technologies  reach  the  middle  stages,  the  only  readily 
available  source  of  funding  is  through  an  acquisition  effort.  The  base  case  policy  is  also  fairly 
similar  to  the  risk  mitigation  strategy  that  many  acquisition  programs  use:  try  to  utilize  the  most 
promising  technology,  but  if  that  fails,  fall  back  to  the  existing,  mature  technology.  Thus,  it 
would  seem  that  given  the  circumstances  that  most  acquisition  programs  operate  under,  the 
business  as  usual  policy  is  quite  rational.  Of  course,  it  should  be  pointed  out  that  all  of  the 
acquisition  policies  perform  better  when  middle-stage  R&D  is  well  funded. 

The  final  scenario  represents  the  impact  of  production  costs  on  the  affordability  of 
evolutionary  acquisition.  The  production  cost  rate  was  varied  from  zero  to  $8  billion  per  year. 
Figure  7  reveals  that  as  procurement  cost  increases,  the  spread  between  the  operating  costs  of 
the  three  policies  increases.  The  shorter  the  acquisition  cycle,  the  more  frequently  production 
costs  are  incurred  and,  consequently,  the  greater  the  impact  of  an  increase  in  production  costs. 
Conversely,  the  lower  production  costs  are,  the  more  cost  effective  evolutionary  acquisition 
becomes. 
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Figure  7.  The  Annual  Acquisition  System  Operating  Cost  as  a  Function  of  the  Production 

Cost  Rate 


6.  Discussion 

The  production  cost  scenario  raises  several  issues  regarding  evolutionary  acquisition. 
Clearly,  the  more  expensive  it  is  to  produce  and  deploy  the  next  iteration  of  a  system,  the  less 
affordable  evolutionary  acquisition  becomes.  But,  of  course,  that  is  dependent  upon  the  nature 
of  the  system  under  consideration,  and  this  is  a  key  difference  between  evolutionary  practices  in 
a  commercial  setting  versus  a  defense  setting.  A  commercial  firm  does  not  purchase  its  own 
product.  In  fact,  if  we  take  the  example  of  a  car  manufacturer — there  is  always  a  substantial 
portion  of  the  customer  base  that  is  looking  to  buy  a  new  car.  Thus,  the  car  manufacturer  is 
going  to  build  and  sell  cars  continuously.  The  costs  of  upgrading  a  model  might  include  the 
costs  of  any  technology  development,  the  cost  of  changing  the  design,  and  the  cost  of  any 
retooling  that  must  be  done  at  production  facilities.  If  the  manufacturer  is  particularly 
successful,  it  may  gain  market  share  from  its  competitors,  and  thus,  the  investment  pays  for 
itself.  Consequently,  a  commercial  firm  can  actually  make  more  money  from  cycling  faster  and 
using  an  evolutionary  approach.  When  the  DoD  would  like  to  buy  a  new  weapon  system,  it 
must  pay  for  all  of  the  same  development  costs  and  purchase  the  product.  Furthermore,  if 
through  more  rapid  acquisition  cycles  the  DoD  improves  the  performance  growth  rate  of  its 
systems,  it  may  outperform  its  adversaries,  but  it  does  not  generate  a  monetary  return  to  help 
fund  the  faster  pace  of  system  development. 

Thus,  the  cost  of  evolutionary  acquisition  is  critically  dependent  upon  the  length  and  cost 
of  stages  in  the  system  acquisition  lifecycle.  The  simulation  model  presented  in  this  paper  was 
generic  in  the  sense  that  it  assumed  that  something  was  acquired  in  each  cycle,  but  it  did  not 
differentiate  between  a  new  system  design  or  a  product  upgrade.  Representing  either  case 
could  be  achieved  by  simply  changing  the  cost  and  duration  parameters  in  the  model.  The  key 
outcome  of  the  evolutionary  policy  was  that  simply  employing  mature  technology  shortened  the 
acquisition  cycle  and  reduced  the  cost  of  each  cycle.  In  the  examples  above,  however,  the 
decline  in  cycle  costs  from  more  efficient  technology  development  alone  was  not  sufficient  to 
compensate  for  the  increase  in  the  cycle  rate.  Thus,  total  acquisition  costs  rose  with 


n.-.i'.uMiAftr.  v  hsiu,, 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  - 121  - 


evolutionary  acquisition.  Some  have  suggested,  however,  that  the  length  and  cost  of  other 
phases  of  the  acquisition  lifecycle  would  decline  under  evolutionary  acquisition  as  well.  The  idea 
is  that  if  acquisition  programs  are  less  ambitious  and  shorter,  development  will  be  easier  and 
there  will  be  fewer  problems  with  unstable  funding.  Thus,  we  should  expect  lower  system 
development  and  procurement  costs  as  well.  Consequently,  the  question  becomes,  if  the  costs 
of  system  development  and  production  decline  under  evolutionary  acquisition,  does  evolutionary 
acquisition  then  become  less  expensive  than  more  traditional  methods? 

To  consider  this  question  let  us  develop  a  very  simple  model  for  the  cost  of  operating  the 
defense  acquisition  system.  First,  we  define  the  following  symbols: 

•  r,j  =  the  acquisition  cycle  rate  for  system  /  under  policy  j  in  cycles  per  year. 

•  Cjj  =  the  cost  per  acquisition  cycle  for  system  /  under  policy  j. 

•  Kj  =  the  total  cost  per  year  for  operating  the  defense  R&D  system  under  policy  j. 

•  Aj  =  the  annual  cost  of  operating  the  defense  acquisition  system  under  policy  j. 


We  can  define  the  cost  rate  to  operate  the  acquisition  system  under  policy  j  as 

ai  =  Y/,c„+ki 
1=1 

where  n  is  the  number  of  systems  begin  acquired.  Thus,  if  policy  e  represents 
evolutionary  acquisition  and  policy  t  represents  traditional  acquisition,  then  evolutionary 
acquisition  would  be  less  expensive  if  Ae  <  At.  For  the  moment,  let  us  assume  that  all  systems 
being  acquired  have  identical  cost  and  cycle  rates.  This  leaves  us  with  the  relationship 

nreCe  +  Ke  <  nrtC,  +  Kt. 

Furthermore,  if  we  assume  that  we  keep  our  R&D  budget  fixed  we  can  simplify  even 
further  to  yield 


Of  course,  since  the  rate  of  acquisition  is  slower  under  the  traditional  acquisition  policy, 
the  right-hand  side  will  be  strictly  less  than  one.  This  implies  that  a  simple  decline  in  program 
costs  from  evolutionary  acquisition  is  not  sufficient  to  reduce  the  total  cost  to  operate  the 
acquisition  system.  Instead,  program  costs  must  decline  sufficiently  to  offset  the  increase  in  the 
rate  of  acquisition. 

To  better  illustrate  this  point,  imagine  that  acquisition  cycles  were  weekly  and  cost  $10. 
The  operating  cost  would  be  $10  per  week.  Now  let  us  assume  that  we  institute  a  new  policy 
that  reduces  cycle  costs  to  $8  per  cycle,  but  the  cycles  now  occur  twice  as  fast.  That  means 
that  under  the  new  policy,  the  operating  cost  would  be  $1 6  per  week.  Thus,  even  though  the 
cost  per  cycle  decreased,  the  total  cost  increased. 

When  we  consider  defense  acquisition  cycles,  if  the  system  development  and 
procurement  costs  also  drop  under  evolutionary  acquisition,  it  might  seem  to  suggest  that  we 
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could  overcome  this  deficit.  If,  however,  the  durations  of  system  development  and  technology 
development  also  decrease,  then  the  equivalent  cost  threshold  becomes  even  more  difficult  to 
reach.  Furthermore,  if  we  consider  spiral  development  when  there  are  several  short, 
overlapping  cycles,  we  see  that  we  would  require  fairly  low  development,  production,  and 
deployment  costs  to  compensate  for  the  speed  of  the  cycles. 

Thus,  the  critical  question  becomes,  how  does  evolutionary  acquisition  affect  the  length 
and  cost  of  development  and  procurement  activities  versus  a  traditional  single-step  to  capability 
approach?  This  is  not  a  trivial  question,  and  the  answer  will  likely  depend  on  the  type  of  system 
being  acquired.  Upgrades  to  complex,  integrated  systems  can  lead  to  substantial  design 
modifications  to  accommodate  even  seemingly  simple  changes  and  using  more  mature 
technologies  does  not  correlate  to  easier  integration  (Smaling  &  de  Week,  2007).  In  fact, 
experiences  at  Westland  Helicopters  indicate  that  even  when  a  system  such  as  a  military 
helicopter  is  designed  with  modularity  and  upgradeability  in  mind,  changes  can  unexpectedly 
propagate  through  large  portions  of  the  system  design  (Clarkson,  Simons  &  Eckert,  2004; 

Eckert,  Clarkson  &  Zanker,  2004).  At  the  other  end  of  the  spectrum,  systems  with  very  loose 
coupling  between  system  components  may  be  quite  amenable  to  rapid  upgrade  and  change. 
Perhaps  the  most  extreme  example  of  this  type  of  system  is  the  Internet,  in  which  the  system 
architecture  changes  continuously  without  any  supervision  or  control. 

Thus,  this  issue  merits  substantial  additional  research  and  is  really  the  determining 
factor  regarding  evolutionary  acquisition's  potential  for  cost  savings.  This  is  not  to  suggest  that 
if  the  costs  of  acquiring  a  particular  system  type  do  not  decline  under  evolutionary  acquisition 
that  the  approach  is  useless.  The  results  of  this  study  suggest  that  evolutionary  acquisition 
delivers  other  benefits  such  as  a  boost  in  the  capability  of  systems  actually  deployed  in  the  field. 
Instead,  it  simply  means  that  additional  capability  will  continue  to  come  at  additional  cost. 
Consequently,  cost  and  performance  may  be  traded  off  by  simply,  appropriately  spacing 
acquisition  cycles. 

7.  Conclusions  and  Further  Research 

The  results  from  this  simulation  study  lead  to  some  highly  suggestive  findings  and  critical 
avenues  for  future  research.  First  and  foremost,  with  a  first-order  representation  of  the 
acquisition  system,  the  results  suggest  that  the  adoption  of  evolutionary  acquisition  policies  has 
the  potential  to  improve  the  performance  of  deployed  systems.  However,  lower  operating  costs 
for  the  defense  acquisition  system  are  not  automatic.  While  each  individual  program  should  be 
less  expensive  under  evolutionary  acquisition  policies,  the  faster  acquisition  cycle-time  means 
that  development,  production,  and  deployment  costs  are  incurred  more  frequently.  This  may 
overwhelm  any  cost  savings  from  managing  technology  development  more  efficiently.  As 
discussed  in  Section  6,  these  cycle  costs  must  decline  sufficiently  under  evolutionary  acquisition 
to  achieve  net  cost  savings.  Thus,  depending  on  the  type  of  system  being  acquired, 
evolutionary  acquisition  may  actually  be  more  expensive  than  traditional  means  of  acquiring 
military  systems.  This  is  a  critical  issue  for  future  research.  However,  this  should  not  be 
interpreted  as  an  endorsement  of  traditional  acquisition  methods.  Instead,  acquisition  cycle¬ 
time  can  be  used  to  control  the  costs  of  an  evolutionary  policy  without  reverting  to  a  traditional 
approach  that  employs  immature  technology.  A  requirement  for  mature  technologies  can  be 
consistently  imposed  with  the  next  acquisition  cycle  beginning  only  when  it  is  affordable. 

There  are  some  important  caveats  on  this  conclusion,  however.  First,  the  above  results 
are  more  significant  for  military-specific  technologies  than  commercial  technologies. 

Commercial  technologies  will  continue  to  develop  and  improve  regardless  of  the  actions  of  the 
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DoD  because  the  DoD  is  actually  a  small  player  in  the  market.  One  example  is  microprocessor 
technology.  On  the  Comanche  helicopter  program,  the  mission  processing  technology  was 
changed  three  times  because  Intel  introduced  newer  processor  models  faster  than  the  DoD 
could  develop  an  advanced  combat  helicopter  (Rogers  &  Birmingham,  2004).  For  military- 
specific  technologies,  however,  forward  progress  is  dependent  upon  actually  testing  and  fielding 
a  technology  and  gathering  user  feedback.  Thus,  the  faster  acquisition  cycles  are,  the  faster 
learning  can  be  incorporated  into  new  technologies  under  development.  Of  course,  faster 
acquisition  cycles  also  mean  that  exogenously  developed  commercial  technologies  can  also  be 
moved  into  the  field  faster. 

Second,  evolutionary  acquisition  policies  do  not  function  well  when  the  R&D  process  is 
underfunded.  Evolutionary  acquisition  depends  on  a  steady  stream  of  mature  technologies. 
When  the  research  pipeline  is  “starved,”  not  only  does  the  performance  of  deployed  systems 
decline  on  average,  but  it  also  becomes  more  unpredictable.  More  traditional  acquisition 
methods  mitigate  this  risk  by  using  an  acquisition  effort  to  secure  funding  for  technology 
development. 

Third,  the  underfunding  of  middle-stage  technologies,  as  is  typical  for  government 
technology  development  (Cornford  &  Sarsfield,  2004),  also  adversely  impacts  evolutionary 
acquisition  policies.  Under  these  circumstances,  traditional  approaches  to  acquisition  are 
actually  superior  to  evolutionary  methods  since  they  mitigate  the  risk  of  technologies  failing  to 
cross  the  chasm.  Thus,  it  would  seem  that  business  as  usual  is  quite  reasonable  under  the 
current  funding  environment  for  military  R&D  activities.  Though,  it  is  important  to  point  out  that 
traditional  acquisition  policies  under  this  scenario  still  underperform  evolutionary  policies  when 
R&D  is  fully  funded. 

Finally,  there  are  several  features  of  the  current  defense  acquisition  system  that  were 
not  considered  in  this  analysis.  First  and  foremost  among  these  is  concurrency.  For  major 
acquisition  efforts  there  is  often  substantial  overlap  between  the  technology  development, 
system  development,  and  production  phases.  While  this  is  often  an  attempt  to  compress  an 
otherwise  long  acquisition  cycle,  the  resulting  rework  often  increases  costs  and  leads  to 
performance  shortfalls.  This  problem  has  been  extensively  documented  elsewhere,  and  there  is 
no  need  for  it  to  be  recapitulated  here.  If,  however,  the  imposition  of  evolutionary  acquisition  and 
its  shorter  acquisition  cycles  reduced  the  temptation  to  use  a  concurrent  acquisition  strategy,  it 
is  possible  that  there  could  be  a  net  cost  savings  through  the  reduction  of  rework,  but  that 
determination  must  be  relegated  to  future  work.  Other  features  of  defense  acquisition  not 
considered  in  this  model  are  operations  and  maintenance  costs,  basic  research  funding,  non- 
centralized  acquisition  management,  program  cancellation,  program  budgeting,  the  capacity  of 
the  industrial  base,  the  capacity  of  the  government  to  consume,  and  system  integration  issues. 
Each  of  these  factors  certainly  influences  the  behavior  and  cost  effectiveness  of  the  defense 
acquisition  system  and  may  be  examined  in  future  work. 

What  we  can  ultimately  derive  from  this  study  is  that,  at  least  to  a  first  order,  there  are 
definite  benefits  to  the  better  management  and  development  of  new  technologies  implied  in 
evolutionary  acquisition.  A  well-managed  technology  portfolio  leads  to  the  development  of 
technology  options,  which  creates  the  flexibility  to  maximize  the  ability  of  acquired  systems  to 
meet  emerging  threats.  Traditional  programs,  through  their  early  commitment  to  particular 
approaches  and  technologies,  sacrifice  some  of  this  flexibility.  The  outstanding  question  raised 
is  whether  the  increased  flexibility  created  by  evolutionary  acquisition  comes  at  additional  cost. 
What  this  study  revealed  is  that  net  cost  savings  are  not  automatic.  Additional  research  is 
required  to  determine  under  what  circumstances  they  are  possible. 
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•  Traditional  acquisition  programs  utilize  promising  but 
immature  technology, 

•  The  use  of  immature  technology  can  add  substantial  risk 
to  an  acquisition  program  that  tends  to  increase  its  cost 
and  duration. 

•  Evolutionary  approaches  to  acquisition  emphasize  the 
use  of  mature,  proven  technologies  to  shorten  and 
reduce  the  cost  of  acquisition  cycles. 
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•  Thus,  evolutionary  vs.  traditional  acquisition  reduces  to 
the  question  of  how  mature  should  a  technology  be 
when  a  commitment  is  made  to  incorporate  it  into  a 
system  design? 

•  Does  an  evolutionary  technology  policy  increase  the 
performance  and  reduce  the  cost  of  operating  the 
defense  acquisition  system? 

•  To  address  these  questions,  a  discrete  event  simulation 
was  developed  to  model  both  a  technology  development 
process  and  the  defense  acquisition  life-cycle. 
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Approach 
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•  One  can  consider  the  output  of  an  R&D  process  to  be 
technology  options. 

-  Developed  technologies  provide  the  option  to  incorporate  new 
capabilities  into  a  system. 

•  Early  commitment  to  a  technology  inherently  limits  future 
design  options. 

•  Late  exercise,  on  the  other  hand,  maximizes  flexibility, 
but  also  incurs  costs  to  develop  and  maintain  technology 
options. 

•  We  could  characterize  traditional  acquisition  as  the 
former  and  evolutionary  acquisition  as  the  latter. 
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•  Thus,  the  key  issue  is  the  level  of  maturity  at  which  a 
technology  is  selected  for  use  in  acquisition  program. 

•  To  understand  this  issue  it  is  necessary  to  model  the 
linkage  between  technology  development  and  the 
acquisition  life  cycle. 

•  A  discrete  event  simulation  was  developed  in  Arena  10.0 
that  models  both  the  flow  of  new  technologies  through  a 
maturation  process  as  well  as  the  flow  of  acquisition 
programs  through  their  life-cycles. 
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•  The  simulation  of  the  acquisition  system  was  deliberately 
scaled  down  and  idealized. 

•  It  is  intended  to  model  the  “physics”  of  the  defense 
acquisition  system. 

•  This  was  both  to  improve  tractability  and  allow  for 
imperfections  to  be  added  selectively  to  isolate  their 
effects. 

•  The  model  consists  of  three  parts; 

-  A  staged  model  of  technology  development. 

-  A  model  of  the  acquisition  life-cycle. 

-  A  model  of  technical  progress. 
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•  The  key  decision  variable  was  the  minimum  level  of 
maturity  at  which  an  acquisition  program  may  utilize  a 
particular  technology. 

•  Three  main  cases  were  considered 

-  Base  Case:  Min  TRL  =  4,  Fallback  TRL  =  7 

-  Evolutionary:  Min  TRL  =  7 

-  Revolutionary:  Min  TRL  =  4,  Fallback  TRL  =  4 

•  Additionally,  extensive  sensitivity  analyses  were 
performed  on  the  model  parameters. 

•  The  warm-up  period  was  50  years,  and  statistics  were 
collected  over  a  150  years. 

•  There  were  40  replications  for  each  experiment. 
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•  Why  does  this  happen? 

•  Imagine  you  had  one  week  cycles  that  cost  $10  per 
cycle.  If  you  were  able  to  speed  up  the  cycles  to  twice 
per  week  and  cut  costs  to  $8  per  cycle,  each  individual 
cycle  is  cheaper,  but  the  total  cost  would  be  $16  per 
week. 

•  The  reason  is  that  there  is  a  certain  amount  of  overhead 
in  terms  of  design,  production,  and  deployment.  If  this 
overhead  is  not  reduced  sufficiently,  faster  cycles  simply 
increase  costs. 
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•  If  evolutionary  acquisition  is  more  expensive,  then  how  is 
it  that  commercial  firms  have  employed  evolutionary 
approaches  so  successfully? 

•  Commercial  firms  sell  products  in  competitive  markets. 
Rapid  improvements  to  products  often  cost  more  than 
leaving  products  unchanged.  However,  moving  superior 
products  to  market  faster  than  competitors  allows 
commercial  firms  to  maintain  or  increase  sales. 

•  The  DoD  does  not  sell  national  defense.  Outcompeting 
an  adversary  does  result  in  additional  revenue  to 
compensate  for  the  additional  cost. 
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•  The  results  presented  are  still  dependent  on  some 
particular  modeling  assumptions. 

•  The  simulation  treated  acquired  systems  generically. 

•  We  hypothesize  that  the  cost  effectiveness  of 
evolutionary  acquisition  will  be  heavily  dependent  upon 
the  type  of  system  being  acquired. 

-  E.g.,  aircraft,  ships,  software  systems,  electronic  systems,  etc. 

•  More  specifically,  systems  that  are  either  more  modular 
in  design  or  are  acquired  in  large  numbers  will  be  more 
amenable  to  evolutionary  acquisition  than  systems  that 
are  tightly  integrated  and  acquired  in  small  numbers. 
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•  Comparative  performance 

•  Requirements  evolution 

•  Rework  propagation 

•  Concurrency 

•  R&D  portfolio  management 

•  Architectural  modularity 
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Output 

Base  Case 
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Revolutionary 
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Operating  Cost 
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Capability  Growth  Rate  vs  R&D  budget 
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Capability  Growth  Rate  vs  Stage  Length 
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Sensitivity  of  Annual  Cost  to  Staging  Probabilities 
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Sensitivity  of  Annual  Cost  to  Stage  Costs 
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